2019-05-24
make scrapy requests from subfunction
stackoverflow
Question

If I make scrapy requests from a subfunction I get this error:

ERROR: Spider must return Request, BaseItem, dict or None, got 'generator'

Example code:

def parse(self):
    yield self.subfunc(link)

def subfunc(self):
    yield scrapy.Request(link)

I assume it has to do with the way generators work but I am not really familiar with them. Any quick fix? I would like to use the subfunction to organise the code a little bit. I am writing to a csv myself so I do not need to yield items or dicts

Answer
1

Your code yields the generator, you want to yield requests from that generator instead.
Probably the best way to do that is using yield from:

def parse(self):
    yield from self.subfunc(link)

Edit:

Since yield from syntax is new in python 3.3, on older versions you'll have to loop over the values yielded by the generator:

def parse(self):
    for request in self.subfunc(link):
        yield request
make scrapy requests from subfunction
See more ...