Python抓取IMDB电影Top250排行

用Scrapy抓取IMDB电影Top250排行


scrapy主代码:

# 抓取主代码
import scrapy
import re
from douban.items import ImdbTopItem
class Imdb(scrapy.Spider):
def __init__(self):
self.start = 0
name = "imdb"
allowed_domains = ["imdb.com"]
start_urls = [
'http://www.imdb.com/chart/top', # imdb top250
]
def parse(self, response):
item = ImdbTopItem()
res_infos = response.css('tbody.lister-list tr')
for res in res_infos:
tmp = {}
title = res.css('td.titleColumn a::text').extract_first() # 电影名称
item['title'] = title
score_and_count = res.css('td strong').extract_first() # 电影评分和评分人数
score_and_count = score_and_count.split('"')[1].split()
score = score_and_count[0]
count = score_and_count[3].replace(',','')
item['score'] = float(score)
item['count'] = int(count)
yield item
# 将结果写入markdown
class ImdbTopPipeline:
def __init__(self):
self.id = 1
def process_item(self, item, spider):
with open('imdb250.md', 'a+', encoding='utf-8') as f:
title = item.get('title')
score = item.get('score')
count = item.get('count')
f.write('|' + str(self.id) + '|' + title + '|' +
str(score) + '|' + str(count) + '|\n')
self.id += 1
return item

结果如下:

排名 电影名 评分 评分人数
1 The Shawshank Redemption 9.2 1883506
2 The Godfather 9.2 1285845
3 The Godfather: Part II 9.0 886997
4 The Dark Knight 9.0 1859466
5 12 Angry Men 8.9 516568
6 Schindler’s List 8.9 968252
7 Pulp Fiction 8.9 1473311
8 The Lord of the Rings: The Return of the King 8.9 1345966
9 The Good, the Bad and the Ugly 8.8 557960
10 Fight Club 8.8 1509622
11 The Lord of the Rings: The Fellowship of the Ring 8.8 1366216
12 Forrest Gump 8.7 1421157
13 Star Wars: Episode V - The Empire Strikes Back 8.7 939935
14 Inception 8.7 1648537
15 The Lord of the Rings: The Two Towers 8.7 1218278
16 One Flew Over the Cuckoo’s Nest 8.7 751948
17 Goodfellas 8.7 811598
18 The Matrix 8.7 1354909
19 Seven Samurai 8.6 253772
20 Star Wars: Episode IV - A New Hope 8.6 1012196
21 City of God 8.6 586293
22 Se7en 8.6 1148671
23 The Silence of the Lambs 8.6 1002869
24 It’s a Wonderful Life 8.6 311299
25 Life Is Beautiful 8.6 483144
26 The Usual Suspects 8.6 823441
27 Léon: The Professional 8.5 817242
28 Saving Private Ryan 8.5 991010
29 Spirited Away 8.5 483881
30 American History X 8.5 866809
31 Once Upon a Time in the West 8.5 242090
32 Interstellar 8.5 1115165
33 The Green Mile 8.5 891944
34 Psycho 8.5 475278
35 Casablanca 8.5 428832
36 City Lights 8.5 125253
37 The Intouchables 8.5 588679
38 Modern Times 8.5 164584
39 The Pianist 8.5 565955
40 Raiders of the Lost Ark 8.5 731134
41 The Departed 8.5 972961
42 Rear Window 8.5 354477
43 Terminator 2 8.5 822315
44 Back to the Future 8.5 827130
45 Whiplash 8.5 514490
46 Coco 8.5 25147
47 Gladiator 8.5 1092781
48 The Prestige 8.5 956135
49 The Lion King 8.5 733203
50 Memento 8.5 939865
51 Apocalypse Now 8.5 496435
52 Alien 8.4 641394
53 The Great Dictator 8.4 157296
54 Sunset Boulevard 8.4 157002
55 Cinema Paradiso 8.4 174140
56 Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb 8.4 378197
57 The Lives of Others 8.4 289342
58 Grave of the Fireflies 8.4 163767
59 Paths of Glory 8.4 136604
60 Django Unchained 8.4 1084200
61 The Shining 8.4 690083
62 WALL·E 8.4 809701
63 American Beauty 8.4 909272
64 Blade Runner 2049 8.4 155188
65 The Dark Knight Rises 8.4 1266344
66 Princess Mononoke 8.4 256234
67 Oldboy 8.4 406075
68 Aliens 8.4 546166
69 Witness for the Prosecution 8.4 75374
70 Once Upon a Time in America 8.4 246744
71 Das Boot 8.3 186637
72 Dangal 8.3 76616
73 Citizen Kane 8.3 328467
74 Vertigo 8.3 281975
75 North by Northwest 8.3 243021
76 Star Wars: Episode VI - Return of the Jedi 8.3 770814
77 Braveheart 8.3 815831
78 Reservoir Dogs 8.3 743881
79 M 8.3 111422
80 Requiem for a Dream 8.3 637667
81 Like Stars on Earth 8.3 112647
82 Amélie 8.3 593446
83 Your Name 8.3 63763
84 A Clockwork Orange 8.3 622022
85 Lawrence of Arabia 8.3 215076
86 Amadeus 8.3 302377
87 Double Indemnity 8.3 110521
88 Eternal Sunshine of the Spotless Mind 8.3 744487
89 Taxi Driver 8.3 569837
90 To Kill a Mockingbird 8.3 242291
91 Full Metal Jacket 8.3 544311
92 2001: A Space Odyssey 8.3 480379
93 Singin’ in the Rain 8.3 172244
94 Toy Story 8.3 702776
95 3 Idiots 8.3 255546
96 The Sting 8.3 195717
97 Toy Story 3 8.3 610359
98 Inglourious Basterds 8.3 1001114
99 Bicycle Thieves 8.3 108478
100 The Kid 8.3 80374
101 Dunkirk 8.3 238681
102 Snatch 8.3 661292
103 Monty Python and the Holy Grail 8.3 418666
104 Good Will Hunting 8.3 684877
105 The Hunt 8.3 205230
106 For a Few Dollars More 8.3 179233
107 L.A. Confidential 8.3 453045
108 Scarface 8.3 598115
109 The Apartment 8.2 124142
110 Metropolis 8.2 125907
111 A Separation 8.2 172584
112 Rashomon 8.2 116212
113 Indiana Jones and the Last Crusade 8.2 572678
114 All About Eve 8.2 93438
115 Yojimbo 8.2 83346
116 My Father and My Son 8.2 51423
117 Up 8.2 752577
118 Batman Begins 8.2 1087175
119 Some Like It Hot 8.2 195530
120 The Treasure of the Sierra Madre 8.2 88551
121 Unforgiven 8.2 307704
122 Downfall 8.2 275391
123 Die Hard 8.2 653600
124 Raging Bull 8.2 259469
125 Heat 8.2 475184
126 Children of Heaven 8.2 38700
127 The Third Man 8.2 127124
128 The Great Escape 8.2 183675
129 Ikiru 8.2 48384
130 Chinatown 8.2 239930
131 Pan’s Labyrinth 8.2 517626
132 My Neighbor Totoro 8.2 203425
133 Incendies 8.2 100797
134 Ran 8.2 86683
135 Judgment at Nuremberg 8.2 51542
136 The Gold Rush 8.2 77072
137 The Secret in Their Eyes 8.2 151886
138 Inside Out 8.2 448968
139 Howl’s Moving Castle 8.2 246295
140 On the Waterfront 8.2 113058
141 The Bridge on the River Kwai 8.2 166152
142 The Seventh Seal 8.2 125451
143 Room 8.2 249020
144 Lock, Stock and Two Smoking Barrels 8.2 454573
145 Mr. Smith Goes to Washington 8.2 85651
146 A Beautiful Mind 8.2 690397
147 Casino 8.2 371933
148 Blade Runner 8.2 546865
149 The Elephant Man 8.1 180303
150 V for Vendetta 8.1 875980
151 The Wolf of Wall Street 8.1 908894
152 Wild Strawberries 8.1 71244
153 The General 8.1 60629
154 Warrior 8.1 368958
155 Dial M for Murder 8.1 121836
156 Trainspotting 8.1 536901
157 Gran Torino 8.1 615868
158 Gone with the Wind 8.1 237987
159 Sunrise 8.1 34414
160 The Deer Hunter 8.1 257214
161 The Sixth Sense 8.1 774401
162 Fargo 8.1 506502
163 The Thing 8.1 292383
164 Andrei Rublev 8.1 28866
165 No Country for Old Men 8.1 687595
166 The Big Lebowski 8.1 599552
167 Finding Nemo 8.1 780772
168 Tokyo Story 8.1 33330
169 The Bandit 8.1 44515
170 There Will Be Blood 8.1 417512
171 Cool Hand Luke 8.1 131964
172 Come and See 8.1 32878
173 Rebecca 8.1 97639
174 Hacksaw Ridge 8.1 279086
175 Kill Bill: Vol. 1 8.1 816589
176 How to Train Your Dragon 8.1 544612
177 Rang De Basanti 8.1 85653
178 The Passion of Joan of Arc 8.1 31159
179 Mary and Max 8.1 134528
180 Gone Girl 8.1 670801
181 Shutter Island 8.1 895878
182 La La Land 8.1 326347
183 Into the Wild 8.1 477567
184 Life of Brian 8.1 299611
185 It Happened One Night 8.1 73281
186 Wild Tales 8.1 121795
187 Logan 8.1 411911
188 Platoon 8.1 320661
189 The Wages of Fear 8.1 40208
190 Hotel Rwanda 8.1 289100
191 Network 8.1 115927
192 In the Name of the Father 8.1 124678
193 Rush 8.1 354519
194 Stand by Me 8.1 302738
195 A Wednesday 8.1 55907
196 Persona 8.1 72052
197 Thor: Ragnarok 8.1 158377
198 Ben-Hur 8.1 180292
199 The Grand Budapest Hotel 8.1 559101
200 The 400 Blows 8.1 79502
201 Memories of Murder 8.1 82043
202 Jurassic Park 8.1 693815
203 12 Years a Slave 8.1 511351
204 Mad Max: Fury Road 8.1 674063
205 Million Dollar Baby 8.1 534442
206 Spotlight 8.1 295435
207 Stalker 8.1 80665
208 The Truman Show 8.1 750289
209 Amores Perros 8.1 189652
210 Butch Cassidy and the Sundance Kid 8.1 168394
211 Hachi: A Dog’s Tale 8.1 189966
212 Nausicaä of the Valley of the Wind 8.1 110737
213 Before Sunrise 8.1 210079
214 The Princess Bride 8.1 324664
215 The Maltese Falcon 8.1 125435
216 Paper Moon 8.0 27559
217 The Nights of Cabiria 8.0 30759
218 Prisoners 8.0 458778
219 Harry Potter and the Deathly Hallows: Part 2 8.0 616742
220 The Grapes of Wrath 8.0 70456
221 Rocky 8.0 423095
222 Catch Me If You Can 8.0 622083
223 Diabolique 8.0 47754
224 Touch of Evil 8.0 80610
225 Gandhi 8.0 189202
226 Monsters, Inc. 8.0 660037
227 Donnie Darko 8.0 642498
228 Barry Lyndon 8.0 114377
229 Annie Hall 8.0 216614
230 The Terminator 8.0 664145
231 Star Wars: The Force Awakens 8.0 699093
232 The Bourne Ultimatum 8.0 542223
233 The Wizard of Oz 8.0 320347
234 Groundhog Day 8.0 489374
235 La Haine 8.0 116956
236 Munna Bhai M.B.B.S. 8.0 56022
237 8.0 87786
238 Jaws 8.0 460322
239 The Best Years of Our Lives 8.0 45083
240 Twelve Monkeys 8.0 505012
241 Infernal Affairs 8.0 101422
242 In the Mood for Love 8.0 88543
243 Paris, Texas 8.0 59236
244 The Help 8.0 354920
245 Beauty and the Beast 8.0 349684
246 Dead Poets Society 8.0 309202
247 The Handmaiden 8.0 48314
248 The Battle of Algiers 8.0 42821
249 Pirates of the Caribbean: The Curse of the Black Pearl 8.0 895005
250 PK 8.0 114037