Custom Metadata - A better way to manage Skuid page/pageassignment/etc. migrations?

Not sure I follow you completely, sorry.  Can you explain step-by-step what you’re thinking?

Pat,

You might want to look at Blue Canvas (https://bluecanvas.io/) as another option for simple source control of Salesforce.

If you want to trigger an external system when a page changes, you can also look at Salesforce’s Platform Events.  They just released this in Summer 17.   It will create an immutable object for each change of any object.  It publishes this change on a ‘bus’ that you can have an external application ‘subscribe to’.  This link has a nice overview:  https://developer.salesforce.com/blogs/developer-relations/2017/05/first-impressions-platform-events…

As far as where you want to go with your setup to manage Skuid page changes in source control, I think you should go down the route of creating a Static Resource per Skuid Page/Assignment/Theme.  This will make it easy to merge individual items if needed and you can granularly move items from Dev to Full to Prod.  This means you’ll need a trigger to create/save the static resource and an invocable action that you can run from a Skuid button to pull all changes from the static resource just moved into a new org.  Note that in my ‘design’ you’ll need to watch the Static Resource for changes and have them committed to Source Control.  This is something that BlueCanvas does out of the box.

Thanks,

Bill

We create a sandbox, branch and when need be a static resource for the Skuid page(s) for each feature using Jira’s issue key to name them all. The process works similarly to Bills, but it doesn’t have the automation. 

Hey Bill. Can I see your setup works? What do you use to commit changes from the sandbox to the feature branch?

Blue Canvas looks nice. Not as far along as Gearset, but the continuous monitoring and deployment functionality could make life much easier.

Anyone care to develop the triggers and invocable methods that Bill suggested? Or at least get us started?

Pat - So at which point do you create the page pack?  How many Skuid pages does your org typically have? I’m still not following the entire process that you’re using, sorry.  Can you walk it through step-by-step?  For example:

1) Feature is requested
2) Issue gets created in JIRA
3) SCRUM occurs and feature gets assigned
4) …

Bill - BlueCanvas looks intriguing, thanks for pointing this out.  It appears that it automates some of the processes that we’ve been discussing here with regards to keeping org & VCS in sync - it’s the build vs. buy decision and I’m always a big fan of “buy” vs “build” :)  On the downside, the challenge with having a static resource for every page/assignment/theme is that you could quickly end up with lots of these.  In the last Skuid app I worked on, we had hundreds of pages.  Do you have any experience with using your ‘design’ in larger orgs with lots of Skuid pages?

Matt - (Note: Friday humor coming in 3…2…1) Probably not what you were thinking when you asked to ‘get us started’ but  here’s a start 🙂

trigger trgPageToVCS on skuid__Page__c (after insert, after update, after delete, after undelete) {<br />&nbsp; &nbsp;&#47;&#47; create static resource for page<br />&nbsp; &#47;&#47;&nbsp; or publish event/make outbound call to 'notify' background process to grab latest page<br />}

Brilliant, @Barry! That’s exactly what I needed! 😉

On a serious note… what 's wrong with having lots of static resources?

Glad you enjoyed Matt :) 

Technically, there’s nothing wrong with lots of static resources per se.  However, the more “stuff” your managing via non-native processes the more risk you introduce in to your overall solution.  Additionally, org refreshes with dev tools, etc. become slower as do deployments (more files to sync) thereby decreasing developer efficiency.  Lastly, there are governors on static resources (no file larger than 5MB & 250MB total).  Technically it can work, it just introduces variables and in-turn risk that would otherwise be unnecessary.

I do understand and appreciate your guys desire to find declarative solutions to manage Skuid Pages with a VCS and there are some solutions that will work.  The main challenge is that there isn’t an available tool today that does exactly what you guys want/need and therefore a custom solution is required to facilitate it.  While this can be done, that solution needs to be maintained & supported which takes away from focusing on running your business and adding value to that business.  Personally, I just think that the increased “risk” and “overhead” associated with any of those processes isn’t worth the “reward.”  That said, happy to help continue to find possible solutions to the problem - i just wouldn’t recommend any of them 🙂

The idea of just notifying the VCS (or some kind of system) to run ‘skuid-pull’ for the changed page seems ideal to me.  Are there servics that could do that kind of thing?

I’m completely out of my depth here. I’ve picked up skuid-grunt and use the CLI for skuid and git with bitbucket (turns out the commands are much faster than using SourceTree).

We’ve been pounding away at trying to automate some kind of deployment system that would work well for both salesforce metadata and skuid data.

Let me try to come up with a fancy picture to demonstrate what our system looks like, and then maybe you folks can offer some suggestions?

That would be very helpful Matt.  The more details on what you guys currently have process & tools wise the more meaningful our discussions around them can be.

Regarding just notifying the VCS to run the CLI commands, if it were me and I was absolutely forced to find a declarative solution, I would target the absolute minimum to get the job done.  This would mean, as you suggest, finding a simple way to initiate a “skuid-pull/push/etc.” when a page is changed.  In my other post, I mentioned creating a GUI wrapper on top of the CLI tools or a browser extension - either of these would probably be where I’d start.  This avoids having to do anything on the SFDC side (e.g. triggers, events, etc.) and instead, just requires the developer to remember to push the button in the GUI when they are done developing their page.  The GUI could use SFDC apis and SOQL to query for information, pages, etc. and present the user with a drop down that allows them to “pick” which page they want to “commit to VCS”. The browser extension could just take the current page information and initiate the CLI commands although this gets a little complicated due to security limitations from within a browser.

Ok, here’s approximately what we’re doing at the moment…
It’s a manually intensive process, which is mostly working, but we run into hiccups primarily with merging SFDC metadata between orgs after a git merge.

We were attempting to use grunt to create an automated deployment process–basically looping through all our customers and running all the commands that we run on the command line in the right sequence–but it’s fraught with opportunities for failure.

Suggestions?

We don’t necessarily need a clicks-only solution (although it would make it easier for the over Dev on my team). I’m mostly interested in finding a way to better streamline or automate the deployment process.

Barry,

Nice start…where’s the Github/Gitlab/Bitbucket repo???  🙂

Unfortunately, I don’t have experience using my proposed design.  I was ‘threading’ off a Pat’s design and adding some options to the mix.  I have used Skuid Grunt, but primarily for page backup and check in to Git.  I have had the same thoughts as Pat in that there ‘ought to be a better way’.  I just haven’t pursued them in earnest.

I think BlueCanvas has the nicest out of box experience for declarative and Apex developers.  It keeps track of everything with limited fuss.

Thanks,

Bill

So many ways to do this! I already know what Skuid likely does with their own org since they’ve got engineers up the wazoo.

Assuming no failed tests, our process is as follows.

1. Start new feature.
   a) Create Page Pack with needed pages named as per Jira issue key.
   b) Create new sandbox named as per Jira issue key.
   c) Create branch from Jira off master named as per Jira issue key. (BitBucket integration in Jira)
2. Develop
   a) On dev sandbox
3. Commit to Branch
   a) Update page pack
   a) Compare and Deploy from Sandbox to Branch using Gearset
4. Pull Request
   a) Create Pull Request from Jira. (BitBucket integration in Jira)
5. Code Review and Merge
   a) In BitBucket
6. Continuous Integration
   a) Auto Deployed to partial sandbox for apex tests.
7. Deploy to UAT
   a) Deploy to Full Sandbox using Gearset.
8. Deploy to Production

Workflow in Jira

Diagram of the process

Hey Matt -

Appreciate the pictures, helpful to better understand, thank you!  The one thing I learned from working with SFDC is that managing “source” is extremely challenging and ripe for failure.  You do need to be meticulous in the way that you manage things and unfortunately, the tools available today just aren’t what they need to be.  The introduction of skuid-grunt was very helpful and moved the needle forward but challenges still are present with SFDC metadata and its limitations as well as skuid resources.  In short, as much as SFDC wants everyone to believe that it’s an “enterprise development” platform, in my opinion it’s not.  The tooling just isn’t there - although it’s come a long way in the last year (e.g. VS Code extension, SFDC DX, etc.) and hopefully will continue to get better.

Questions:
1) Can you provide some specifics on what challenges you’re having with merging metadata?  One thing I’ve noticed in the past (although it’s been quite a while) is that if you don’t keep your sandboxes in sync then merges become much more difficult especially around “SFDC Upgrade” time when new features/metadata gets introduced to the orgs.

2) For the automation you do/did have in place, looping through customers, what challenges or opportunities for failure were you facing?

Thanks Bill, all makes sense and agreed - there “should” be a better way 🙂

Hey Pat -

Thanks for this, very helpful.  A few thoughts:

1) Your approach seems risky unless you are very careful about which pages you are modifying in each feature.  For example, what do you do when “Feature A” and “Feature B” are worked on simultaneously and both need to modify “Page ABC”?  Using page packs would make this very difficult to merge properly.

2) You mentioned each page pack has its own “module” name.  Did you mean Skuid “Module”?  If so, how do you reconcile these upon merge to CI/UAT/etc?  In other words, if Feature A gets a module name of “Feature A” then all pages will be in that Module.  What if this feature needs to modify a page that already exists in a different module?  Or what if this feature is creating a new master page or shared page of some type that another feature at some point needs?  Do the pages remain in Feature A Module? 

3) If you were to use the CLI (I understand you don’t want to so just humor me :)), do you think there would be other challenges you’d face if you just swapped in CLI to your process instead of page packs?

4) With your current process, what challenges are you facing specifically?

Never responded to this.

We’ve settled on working on Skuid pages one at a time.

I see challenges in using CLI since most of the developers i work with aren’t coding developers. They never use CLI. They are declarative developers like myself. The Gearset UX is way way easier to learn and execute on.

Page packs are required in order to get the pages into other orgs via source control. Do Skuid Grunt or Skuid CLI tools alleviate this? If so how?

Hey Pat - Yes, Skuid CLI (or Grunt) provide a mechanism to handle individual pages and pull/push to any org.  After “pulling” the pages from the org, you use your VCS of choice to commit them to source control.  See the readme at https://github.com/skuid/skuid for more details on how to use.

I guess what I’m missing is an understanding of being able to view the Skuid page XML as metadata so we can commit, review and merge changes to the page. Such as adding an action to a button. Right now, the only thing I see options for is to replace the page in it’s entirety.

When you “pull” (using Skuid CLI), it will grab the latest XML for the specified resources from the org and place a copy of those on your file system.  As you mention, this will “replace” the copy that you currently have on your file system.  You can then view the XML with any editor (e.g. VS Code, Eclipse, notepad++, vi, etc.) When you commit those changes (typically via a Pull Request if using GIT), you’ll need to merge them.  The “merge” is the same process that would be needed for any type of source controlled file (e.g. sfdc metadata files, js files, etc.).

Yup. Still not clear for me. Is this process going to allow the XML of the page to be updated in the merge? Will the dif show the that on line 68 the URL parameter was changed from “Id” to “id”? If so, what’s the process to subsequently deploy the page from source control to a Salesforce org? As a page pack it goes in with a change set or a deployment in Gearset. Final step would be to unpack the page.

Yes, the diff would show each character/line that changed from what’s in your VCS to what’s being proposed to being committed.  

Deployment to an org would come via an automated release process that uses the CLI.

I’m not familiar with Gearset so not familiar with how it works.  Assuming Gearset uses the local file system to manage pushes/pulls to your org, using the CLI should be no different than managing SFDC metadata or the page packs.  If GearSet doesn’t use the local file system (or doesn’t at least provide the option to use the local file system), then CLI might not be an option.

 If you’re managing your page packs such that each pack contains 1 and only 1 page, then using the CLI won’t buy you much other than alleviating having to manually create packs for each page. 

My best advice here is to try it and see if it yields the results you’re after 🙂

Ok. Seems like everything desired is achieved when deploying from a local system w/ Skuid CLI?

I already know that local file system and CLI is a barrier of entry for most declarative developers. We’re declarative because we don’t like to go deeper into the weeds unless it’s easy. 😉

So everything except easy.

I’ll leave to you to decide if everything is achieved when deploying from local system w/ Skuid CLI since you know your requirements best :)  Without knowing how GearSet works, I can’t be sure but it’s possible you can leverage whatever the process is that GearSet uses and invoke a CLI command from that process.

Regarding not liking to go deeper into the weeds “unless it’s easy” - I’d challenge you on that :)  I recall a time a while back when you dug in to javascript and Skuid API despite no prior knowledge of JS and, unlike using a CLI tool, involved actual coding.  Just like everything else, there’s what you can do out of the box declaratively but to really harness the “power”, sometimes you need to jump in to the deep end.  🙂