web crawling

[simoncpu]

---

naa kay setup ani?

You can just use a normal Unix or Linux box. Here's a sample on how to spider the URLs from PNOY's website:

$ wget -r -l 0 http://president.gov.ph/

$ for i in `egrep -Ihsoiwr '(http[s]*[:][/]+|www[.])[^"\<>]*' * |  egrep -v '^$|^#'^C`; do
       echo "INSERT INTO lestat('url') VALUES('$i');"
  done

Sample output:

INSERT INTO lestat('url') VALUES('http://www.twinhelix.com^M');
INSERT INTO lestat('url') VALUES('http://creativecommons.org/licenses/LGPL/2.1/^M');
INSERT INTO lestat('url') VALUES('http://www.president.gov.ph/images/seal.png');
INSERT INTO lestat('url') VALUES('http://www.facebook.com/pages/Noynoy-Aquino-P-Noy/141976959168393?ref=mf?');
INSERT INTO lestat('url') VALUES('http://twitter.com/#!/PresidentNoy');
INSERT INTO lestat('url') VALUES('http://www.youtube.com/user/RTVMPNoy');
INSERT INTO lestat('url') VALUES('http://api.recaptcha.net/challenge?k=6Lc_TbwSAAAAAFnJFB4XffLPFsftPkyexkr143PJ');
INSERT INTO lestat('url') VALUES('http://api.recaptcha.net/noscript?k=6Lc_TbwSAAAAAFnJFB4XffLPFsftPkyexkr143PJ');
INSERT INTO lestat('url') VALUES('http://www.president.gov.ph');
INSERT INTO lestat('url') VALUES('WWW.PRESIDENT.GOV.PH');
INSERT INTO lestat('url') VALUES('http://gov.ph');
INSERT INTO lestat('url') VALUES('http://www.idocs.com^M');

NOTE: Some of the output are invalid. You still need to fix the regex or add a URL validation to make it more robust.

[ simon.cpu ]

[lestat1116]

then you just data mine...

im looking for program na mo data mine. manual man gud ako gamit. copy stuff den paste sa db.

[salbahis]

im looking for program na mo data mine. manual man gud ako gamit. copy stuff den paste sa db.

hmmmm.... odesk? data entry?

[lestat1116]

hmmmm.... odesk? data entry?

yup data entry but not odesk. naa ka setup?

[salbahis]

yup data entry but not odesk. naa ka setup?

yeah, but its not for sharing its for my wifes own odesk job!!

[ad_dur]

nose bleed

[lestat1116]

yeah, but its not for sharing its for my wifes own odesk job!!

im not in odesk... hehe.. i just need help.

[lestat1116]

up lang nako ni..

[kamahak]

Naay Lucene based crawler ang Apache. Nutch. Taw-anan pero magamit ni nimo. Basaha lng documentation or search for introductory tutorials.

Naa puy Heritix ug OpenSpider.

Wa pko kagamit aning duha, pero maayo mn ug reviews, especially ang Heritix.

[lestat1116]

thanks mura dako jud ako study mode ani.