make scrapy requests from subfunction
源自: stackoverflow

If I make scrapy requests from a subfunction I get this error:

ERROR: Spider must return Request, BaseItem, dict or None, got 'generator'

Example code:

def parse(self):
    yield self.subfunc(link)

def subfunc(self):
    yield scrapy.Request(link)

I assume it has to do with the way generators work but I am not really familiar with them. Any quick fix? I would like to use the subfunction to organise the code a little bit. I am writing to a csv myself so I do not need to yield items or dicts


Your code yields the generator, you want to yield requests from that generator instead.
Probably the best way to do that is using yield from:

def parse(self):
    yield from self.subfunc(link)


Since yield from syntax is new in python 3.3, on older versions you'll have to loop over the values yielded by the generator:

def parse(self):
    for request in self.subfunc(link):
        yield request
make scrapy requests from subfunction