Simple Python Search Spider, Page Ranker, and Visualizer
This is a set of programs that emulate some of the functions of a
search engine. They store their data in a SQLITE3 database named
'spider.sqlite'. This file can be removed at any time to restart the
process.
You should install the SQLite browser to view and modify
the databases from:
http://sqlitebrowser.org/
This program crawls a web site and pulls a series of pages into the
database, recording the links between pages.
Note: Windows has difficulty in displaying UTF-8 characters
in the console so for each console window you open, you may need
to type the following command before running this code:
chcp 65001
http://stackoverflow.com/questions/388490/unicode-characters-in-windows-command-line-how
Mac: rm spider.sqlite
Mac: python3 spider.py
Win: del spider.sqlite
Win: spider.py
Enter web url or enter: http://www.dr-chuck.com/
['http://www.dr-chuck.com']
How many pages:2
1 http://www.dr-chuck.com/ 12
2 http://www.dr-chuck.com/csev-blog/ 57
How many pages:
In this sample run, we told it to crawl a website and retrieve two
pages. If you restart the program again and tell it to crawl more
pages, it will not re-crawl any pages already in the database. Upon
restart it goes to a random non-crawled page and starts there. So
each successive run of spider.py is additive.
Mac: python3 spider.py
Win: spider.py
Enter web url or enter: http://www.dr-chuck.com/
['http://www.dr-chuck.com']
How many pages:3
3 http://www.dr-chuck.com/csev-blog 57
4 http://www.dr-chuck.com/dr-chuck/resume/speaking.htm 1
5 http://www.dr-chuck.com/dr-chuck/resume/index.htm 13
How many pages:
You can have multiple starting points in the same database -
within the program, these are called "webs". The spider
chooses randomly amongst all non-visited links across all
the webs.
If you want to dump the contents of the spider.sqlite file, you can
run spdump.py as follows:
Mac: python3 spdump.py
Win: spdump.py
(5, None, 1.0, 3, u'http://www.dr-chuck.com/csev-blog')
(3, None, 1.0, 4, u'http://www.dr-chuck.com/dr-chuck/resume/speaking.htm')
(1, None, 1.0, 2, u'http://www.dr-chuck.com/csev-blog/')
(1, None, 1.0, 5, u'http://www.dr-chuck.com/dr-chuck/resume/index.htm')
4 rows.
This shows the number of incoming links, the old page rank, the new page
rank, the id of the page, and the url of the page. The spdump.py program
only shows pages that have at least one incoming link to them.
Once you have a few pages in the database, you can run Page Rank on the
pages using the sprank.py program. You simply tell it how many Page
Rank iterations to run.
Mac: python3 sprank.py
Win: sprank.py
How many iterations:2
1 0.546848992536
2 0.226714939664
[(1, 0.559), (2, 0.659), (3, 0.985), (4, 2.135), (5, 0.659)]
You can dump the database again to see that page rank has been updated:
Mac: python3 spdump.py
Win: spdump.py
(5, 1.0, 0.985, 3, u'http://www.dr-chuck.com/csev-blog')
(3, 1.0, 2.135, 4, u'http://www.dr-chuck.com/dr-chuck/resume/speaking.htm')
(1, 1.0, 0.659, 2, u'http://www.dr-chuck.com/csev-blog/')
(1, 1.0, 0.659, 5, u'http://www.dr-chuck.com/dr-chuck/resume/index.htm')
4 rows.
You can run sprank.py as many times as you like and it will simply refine
the page rank the more times you run it. You can even run sprank.py a few times
and then go spider a few more pages with spider.py and then run sprank.py
to converge the page ranks.
If you want to restart the Page Rank calculations without re-spidering the
web pages, you can use spreset.py
Mac: python3 spreset.py
Win: spreset.py
All pages set to a rank of 1.0
Mac: python3 sprank.py
Win: sprank.py
How many iterations:50
1 0.546848992536
2 0.226714939664
3 0.0659516187242
4 0.0244199333
5 0.0102096489546
6 0.00610244329379
...
42 0.000109076928206
43 9.91987599002e-05
44 9.02151706798e-05
45 8.20451504471e-05
46 7.46150183837e-05
47 6.7857770908e-05
48 6.17124694224e-05
49 5.61236959327e-05
50 5.10410499467e-05
[(512, 0.02963718031139026), (1, 12.790786721866658), (2, 28.939418898678284), (3, 6.808468390725946), (4, 13.469889092397006)]
For each iteration of the page rank algorithm it prints the average
change per page of the page rank. The network initially is quite
unbalanced and so the individual page ranks are changing wildly.
But in a few short iterations, the page rank converges. You
should run sprank.py long enough that the page ranks converge.
If you want to visualize the current top pages in terms of page rank,
run spjson.py to write the pages out in JSON format to be viewed in a
web browser.
Mac: python3 spjson.py
Win: spjson.py
Creating JSON output on spider.js...
How many nodes? 30
Open force.html in a browser to view the visualization
You can view this data by opening the file force.html in your web browser.
This shows an automatic layout of the nodes and links. You can click and
drag any node and you can also double click on a node to find the URL
that is represented by the node.
This visualization is provided using the force layout from:
http://mbostock.github.com/d3/
If you rerun the other utilities and then re-run spjson.py - you merely
have to press refresh in the browser to get the new data from spider.js.
"But I have only one disguise in the house--the old one." "How much is it, madame?" "9. Van Hoeyaerden, member of the Town Council. 121 Previous to his forty-ninth year, Plotinus wrote nothing. At that age he began to compose short essays on subjects which suggested themselves in the course of his oral teaching. During the next ten years, he produced twenty-one such278 papers, some of them only a page or two in length. At the end of that period, he made the acquaintance of his future editor and biographer, Porphyry, a young student of Semitic extraction, whose original name was Malchus. The two soon became fast friends; and whatever speculative differences at first divided them were quickly removed by an amicable controversy between Porphyry and another disciple named Amelius, which resulted in the unreserved adhesion of the former to the doctrine of their common master.415 The literary activity of Plotinus seems to have been powerfully stimulated by association with the more methodical mind of Porphyry. During the five years416 of their personal intercourse he produced nineteen essays, amounting altogether to three times the bulk of the former series. Eight shorter pieces followed during the period of failing health which preceded his death, Porphyry being at that time absent in Sicily, whither he had retired when suffering from the fit of depression already mentioned. He did not answer at once, but sat watching the trumpeter come out of the adjutant's office to sound recall. "Yes, she will marry," he agreed; "if no one else marries her, I will. I am as old as her father would have been but it would save telling some fellow about her birth." "You must get Mrs. Landor into the post to-morrow," Cairness said abruptly; "Victorio's band is about." The next day he left for the Circle K Ranch. Lawton did not appear to need help. But he fired a Greaser, nevertheless, and took Cairness on. He seemed to stand in as abject awe of Stone's note as an Arab might have stood of a bit of the black covering of the Kaabah stone. On the 9th of January, a month after their arrival, Lord Derwentwater was impeached of high treason by Mr. Lechmere in a bitter speech in the Commons. Other members, with equal acrimony, followed with impeachments against the Lords Widdrington, Nithsdale, Wintoun, Carnwath, Kenmure, and Nairn. The impeachments were carried up to the House of Lords on the same day, and on the 19th the accused noblemen were brought before the Peers, where they knelt at the bar until they were desired to rise by the Lord Chancellor, when, with the exception of Lord Wintoun, they confessed their guilt, and threw themselves on the mercy of the king. Sentence of death was immediately pronounced on those who had pleaded guilty; and Lord Wintoun was condemned after trial, but several months later he effected his escape from the Tower. Every effort was made to save the prisoners, and they were all reprieved, with the exception of Derwentwater, Kenmure, and Nithsdale. The first two were executed; but the Countess of Nithsdale, being about to take her leave of her husband, contrived, by introducing some friends, to secure his escape in female attire. Q, 200th Ind. Rush. Yours, Jim." "Reckon I can," said Albert good-naturedly, knowing that some day he might want his brother to do the same for him. "Lie there, dog!—there goes one of the accursed council!" and, springing up the step with a giant grasp, he seized the mitred chancellor by the neck, and dragged him forth into the centre of the church. HoME有没有泷泽萝拉的呀
ENTER NUMBET 0016herocean.com.cn
jiediji.com.cn
www.qdfzmall.com.cn
www.vxtxf.net.cn
www.trenso.com.cn
www.shimoo.com.cn
skchain.com.cn
www.rjshjf.com.cn
www.wybgyp.com.cn
inbp.com.cn