We have an incredibly large .csv file (160,000+ rows) that I have manually split into smaller files of +/- 10,000 rows. This files contains all the site members on all the sites in our farm. We are automating the process of notifying the site owners and members that they need to do something about their unused site. We have a data export of the site owners and members, the parse data workflow separates them by site URL and groups them, then updates the master list of sites.
The parse data workflow worked fine for the site owner files with +/- 10,000 rows. However, the member files choke the timer service, freezes the workflow, and produces unreliable results. I think it's because there can be hundreds of site members per site and the workflow stays on one row in the master list churning and churning while it builds the data.
What is the best plan of attack here? Should we split the .csv files into files of less than 10,000 rows? If so, can anyone recommend an efficient way to do that?
Does anyone have any experience with this sort of thing? When I try to manually enter site members into a record in the master list, I get an error telling me I can only add 200 items and it won't let me save. If the list is throwing that error, could that be what is stalling my workflow?
