If I make scrapy requests from a subfunction I get this error:
ERROR: Spider must return Request, BaseItem, dict or None, got 'generator'
def parse(self): yield self.subfunc(link) def subfunc(self): yield scrapy.Request(link)
I assume it has to do with the way generators work but I am not really familiar with them. Any quick fix? I would like to use the subfunction to organise the code a little bit.
I am writing to a
csv myself so I do not need to yield items or dicts
Your code yields the generator, you want to yield requests from that generator instead.
Probably the best way to do that is using
def parse(self): yield from self.subfunc(link)
yield from syntax is new in python 3.3, on older versions you'll have to loop over the values yielded by the generator:
def parse(self): for request in self.subfunc(link): yield request
- Is there a way to get the subscriber count of two particular users of youtube from their beginning of career to the most recent possible?
- Pytorch argsort ordered, with duplicate elements in the tensor
- How I can I lazily read multiple JSON values from a file/stream in Python?
- Pytorch argsort ordered
- Print series of number every 50
- Selecting multiple columns in a pandas dataframe
- FancyImpute installation in Anaconda
- Simplifying a pair of if-else statements around a Flask SqlAlchemy query
- Finding a key recursively in a dictionary
- Convert UUID 32-character hex string into a “YouTube-style” short id and back