神器 Selenium 专治上交所妖蛾子 - Jacky Liu's Blog
神器 Selenium 专治上交所妖蛾子
---- 上交所又出妖蛾子了，前一阵子股票列表查询页面的 url 又不能用，第三次了。
---- 一直以来我都从上交所网站上抓取股票列表信息。网站对应的页面用的是动态技术，网址是由 javascript 生成，对访问者隐藏的。没关系，用抓包分析，就能看见如下一个网址:
http://www.sse.com.cn/sseportal/webapp/datapresent/SSEQueryStockInfoAct?reportName=BizCompStockInfoRpt&PRODUCTID=&PRODUCTJP=&PRODUCTNAME=&keyword=&tab_flg=1&CURSOR={cursor}
{cursor}是替换部分，程序只要替换成每页的起始偏移就能抓取到全部的股票列表。但是过了一年多以后，这个 url 不能用了。
---- 仍然抓包，发现这次访问的 url 是如下形式:
http://query.sse.com.cn/commonQuery.do?jsonCallBack=jsonpCallback51734&isPagination=false&sqlId=COMMON_SSE_ZQPZ_GPLB_MCJS_SSAG_L
这次还真是改进。与前面那个需要翻页不同，这个只要一次访问就返回全部股票列表，两边都省事，而且返回字串是符合 Python 语法的，一个 eval() 函数就能转化成 Python 数据格式。可是好景不长，几个星期之后这个 url 又不能用了。
---- 再来，这回发现 url 改动不大，只是 "jsonpCallback" 后面那一个数字也变成了动态的，每一次访问都不一样。这样一来，除非程序里也包含 javascript 引擎，能根据页面里的 js 代码算出那个值是多少，否则别想获取完整的股票列表了。就算这种方式也只是理论上可以，现实有没有途径还不知道。
---- 这种改动很微妙，目测其目的就是为了防止我这样的人用程序抓取它的页面信息。大概只要你访问了他的网站，他就自动假设你是个小散，而小散当然只配用手一页页翻着看。给机构和服务商都提供有专用的高速数据接口，一年几十万数据费不过湿湿碎而已。相比之下，深交所是把所有股票的代码、简称、公司信息都做成一张表，方便下载。上交所真够有心计。
---- 上交所到底属于什么性质真不好说。不过我总觉得，它多少应该带这么一点公共服务的性质。既然不准各地都建交易所，互相竞争，谁的服务好就到谁那里去上市，既然全国只有两个交易所，上市公司只有两千多家，一个 IPO 券商要收几千万。那么交易所在高价贩卖行情信息之余，稍微附带地给大众提供那么一点免费的服务，比如说查询一下上市股票有多少个，叫什么名字，代码多少之类，应该是理所当然的吧？
---- 既然要提供的话，就像深交所那样，做个列表给大家下载，你的服务器也能减轻负担，因为只要一次访问就成，小散们用着也方便，这也毫不过分吧？
---- 但是现实中奈何不了小人。不仅不提供下载，还想办法禁止老子抓取。禁NMGB。什么 javascript 老子是一概不懂的，就算懂也没功夫跟小人玩猫捉耗子。一通 Google 之后，搜到了一个神器: Selenium。神器是干什么的呢？官网里说的明白:
"Selenium automates browsers. That's it."
在老子看来，就是专门对付小人用的。
---- 代码，不到 60 行就搞定。注意 Selenium 还不支持 py3，这些是 py2 的:
# -*- encoding: utf-8 -*-
import sys
import pickle
import selenium
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
# from selenium.common.exceptions import TimeoutException
# from selenium.webdriver.support import expected_conditions as EC # available since 2.26.0
def wait_condition_01(driver):
return driver.find_element_by_id('dateList_container_pageid')
def extract_table(driver, stocklist):
tag_table= driver.find_element_by_class_name("tablestyle")
tabletext= tag_table.text
stocklist.extend(tabletext.split('\n')[1:])
driver= selenium.webdriver.Firefox()
driver.get("http://www.sse.com.cn/assortment/stock/list/name/")
stocklist= []
extract_table(driver=driver, stocklist=stocklist)
tag_meta= driver.find_element_by_id("staticPagination")
attr_total= int(tag_meta.get_attribute("total"))
attr_pageCount= int(tag_meta.get_attribute("pageCount"))
# 逐页提取内容
for pagenr in range(2, attr_pageCount+1):
id_input= 'dateList_container_pageid' if pagenr > 2 else 'xsgf_pageid'
id_button= 'dateList_container_togo' if pagenr > 2 else 'xsgf_togo'
tag_input= driver.find_element_by_id(id_input)
tag_button= driver.find_element_by_id(id_button)
tag_input.send_keys(str(pagenr))
tag_button.click()
WebDriverWait(driver, 10).until(wait_condition_01)
extract_table(driver=driver, stocklist=stocklist)
# 向主调进程发送结果
data= {
'个股总数': attr_total,
'个股列表': stocklist,
}
driver.quit()
pdata= pickle.dumps(data, protocol=2)
sys.stdout.write( pdata + b'\n' )
---- 这下，除非你上交所的网站彻底不开了。否则老子就是要自动抓取你的股票列表。看着办吧。
2013年9月20日 17:48
你是说下面这个地址是一直在变化的？http://www.sse.com.cn/js/common/ssesuggestdataAll.js
2013年9月21日 23:39
@crosser: 不是，是文里贴的那个网址，中间一部分每次访问都不一样。你的这个网址是怎么得到的。是分析上交所页面的 js 代码看出来的吧？
2013年9月22日 21:35
@Jacky Liu: 是的，不过还是你那种方法比较妥当一些，感谢分享，我才开始学Matplotlib for Python
2013年9月22日 21:46
@crosser: 我那么搞是因为不懂 js。也曾看过网站里的 js 代码，不得要领。只会用用封包分析，封包再不顶用只能另起炉灶了。
matplotlib 很强也好用，祝顺利。
2015年6月02日 00:13
牛人专治小人。:)
2015年9月01日 11:33
callback 的算法倒是简单，但是可能不止这一种，header里面也有限制。
jsonpCallback:"jsonpCallback"+Math.floor(Math.random() * (100000000 + 1)),
2015年9月01日 12:12
@Eric: 不懂 js，不过既然有 random 函数在里面，大概九成是不可做的。
应该就是为了防止机器抓取，真是恶毒。
2022年9月19日 00:13
Geography is an important subject in Class 10t. However, considerable ambiguities persist in its exact position within academia. AP 10th Geography Question Paper As an inclusive discipline, the content of geography is derived from various subjects in the sciences, social sciences and humanities. This is equally true about the geography syllabus in schools as well.Telugu Medium, English Medium & Urdu Medium Students of the State Board can download the AP 10th Geography Model Paper 2023 Pdf with answers for all exam formats conducted by BSEAP for Part-A, Part-B, Part-C, and Part-D exams of SA-1, SA-2, FA-1, FA-2, FA-3, FA-4 along with Assignments which were designed by subject experts of leading educational institutes.
2022年9月27日 15:45
KSEEB Model Paper 2023 Class 5 Pdf Download with Answers for Kannada Medium, English Medium, Hindi Medium, Urdu Medium & Students for Small Answers, Long Answer, Very Long Answer Questions, and Essay Type Questions to Term2 & Term2 Exams at official website.Karnataka Board 5th Class Model Paper New Exam Scheme or Question Pattern for Sammittive Assignment Exams (SA1 & SA2): Very Long Answer (VLA), Long Answer (LA), Small Answer (SA), Very Small Answer (VSA), Single Answer, Multiple Choice and etc.
2023年1月12日 13:33
The Selenium artifact is a powerful tool that is said to control the demonic moths of the Shanghai Stock Exchange. It is believed to have the ability to keep the moths from wreaking havoc on the stock market and knee joint pain causing financial ruin. Many believe that the artifact is a powerful weapon against evil and that it is a must-have for anyone doing business in Shanghai.
2023年9月26日 22:30
Gym body scans are cutting-edge tools that have revolutionized the fitness industry. These scans use advanced technology to provide individuals with a detailed analysis of their body composition, going far beyond the basic measurements of weight and height.
2023年11月09日 14:59
My sincere appreciation for this informative website. Can you suggest other websites that present information in such a clear and concise manner? I'm currently embarking on a new venture and am eager to gather as much knowledge as possible.
2024年1月30日 01:14
In recent years, there has been a growing awareness of the intricate connection between oral health and overall well-being. Numerous studies have highlighted the links between poor oral health and various systemic conditions, including cardiovascular diseases, diabetes, and respiratory infections. The mouth serves as a gateway to the rest of the body, making it imperative to maintain optimal oral hygiene to reduce the risk of systemic health issues. Dentists, therefore, often collaborate with other healthcare professionals to ensure comprehensive patient care.
2024年2月03日 01:07
Dental aesthetics, encompassing procedures like teeth whitening and cosmetic dentistry, acknowledges the impact of a confident smile on an individual's self-esteem and overall quality of life. While aesthetics may be considered elective, the psychological and social benefits of feeling good about one's smile are undeniable. Dental professionals specializing in aesthetic dentistry work to enhance the appearance of teeth, recognizing the symbiotic relationship between oral health and self-confidence.
2024年2月22日 19:31
There is so much in this article that I would never have thought of on my own. Your content gives readers things to think about in an interesting wa