Workflow to Split Large .csv File

We have an incredibly large .csv file (160,000+ rows) that I have manually split into smaller files of +/- 10,000 rows. This files contains all the site members on all the sites in our farm. We are automating the process of notifying the site owners and members that they need to do something about their unused site. We have a data export of the site owners and members, the parse data workflow separates them by site URL and groups them, then updates the master list of sites.

The parse data workflow worked fine for the site owner files with +/- 10,000 rows. However, the member files choke the timer service, freezes the workflow, and produces unreliable results. I think it's because there can be hundreds of site members per site and the workflow stays on one row in the master list churning and churning while it builds the data.

What is the best plan of attack here? Should we split the .csv files into files of less than 10,000 rows? If so, can anyone recommend an efficient way to do that?

Does anyone have any experience with this sort of thing? When I try to manually enter site members into a record in the master list, I get an error telling me I can only add 200 items and it won't let me save. If the list is throwing that error, could that be what is stalling my workflow?

Tags (3)
1 Reply

Re: Workflow to Split Large .csv File


Rather going thru the bulk csv file with list of all sites, its members, follow this approach:

1. Create a site workflow

2. Have your workflow read the CSV file (which consists of site urls)

3. Use a Loop action to read each site url from CSV 

4. With in the loop action, use "Call Web Service" action

5. In "Call Web Service" action, call web service, "http://siteurl/_vti_bin/UserGroup.asmx"

6. Select "GetUserCollectionFromSite" method, more details on this method

7. Capture the returned xml in a variable

8. Use "Query XML" action and get user's email id from attribute, "Email"(the xml structure is shown in the above link)

9. Use "Send Email" action to send email.



0 Kudos