How to work in a collaborative team environment

## How to work in a collaborative team environment ----------- *[2022 MUSES Collaboration Meeting](https://forum.musesframework.io/t/muses-collaboration-meeting-2022)* T. Andrew Manning National Center for Supercomputing Applications @ UIUC --- ### Motivation * **Some of you have not worked in a team before**. Graduate students and postdocs typically work on projects individually, having occasional meetings with their professor to discuss status and next steps. If they work in a team, it is often small and one where day-to-day interation is local and natural. * **MUSES is a large collaboration** between independent research groups from many institutions that are as geographically separated as possible (Japan is ~14 hours ahead of the USA!).  * **Collaborators cannot work in silos** in MUSES, meeting only a few times per year; our work requires frequent communication and coordination to achieve our goals.  * Working together productively and efficiently requires * **agreeing upon communication methods and workflows**, and * **cyberinfrastructure that supports our communication methods and data sharing needs**  ---- ### Overview At NCSA we spend a great deal of effort designing, configuring, and operating the CI for MUSES. It is fairly **cutting-edge in the context of research collaborations**, so there are services and tools you may be unfamiliar with. In this talk I will **introduce these from a practical perspective** by considering some common scenarios we encounter when working together. --- ## Scenarios (Press ESC for an overview) --- ### Asking a question ----------- You have a question about how to install a C++ library for your EoS module. ---- #### Typical approach Compose an email to everyone you think might know the answer, including your advisor, fellow grad student, and a postdoc you work with from another university. ---- #### MUSES approach Create a new topic on the forum in the #general (publicly visible; preferred) or #collaborator (visible only to official MUSES collaborators) category with your question. ---- #### Why Your email will be read by the three original recipients only. If you are lucky, they will know someone with the answer and will Reply-All including that person. After a hopefully productive email exchange of Reply-All emails (with the inevitable loss of some people and addition of others), you will have your answer.  There are many disadvantages to this seemingly successful approach: * If someone else asks your question again, they will not know it was already answered, and the odds are that they will not initally contact the same people you did and may not find the answer. **The forum is *searchable and persistent***.  * You may swing and miss with the people you thought to email, but **on the forum you can reach *everyone* in the collaboration**. Even if no one knows the answer yet, your post will remain visible for others to find and answer at some point.  * Emails are a dead-end. **Topics on the forum can naturally evolve into useful documentation**. Particularly helpful posts or collaboratively composed posts (e.g. meeting notes) can be switched into a "wiki" which will allow anyone to edit its contents. --- ### Sharing files ----------- Your team needs to compile some data files and downloaded academic papers in a common place where everyone can view and edit the files. * Some of the people that need write access are not members of your university, and * there are some people that need to have read-only access that are not even MUSES collaborators. ---- #### Typical approach Create a folder in the Google Drive/Box/OneDrive/Dropbox service licensed by your university. * Share with your team using the native account sharing since everyone has an account through the university. * Share with external collaborators with one write-access link and * the external non-collaborator with a separate read-only link. ---- #### MUSES approach Create a subfolder in [our Nextcloud service within the collaboration-wide shared group folder](https://cloud.musesframework.io/apps/files/). This automatically grants read-write access to collaborators. Generate and share individual read-only links for the external non-collaborator. ---- #### Why * **Flexibility and control**. The MUSES collaboration fully controls our Nextcloud server and data, and so there is no risk that the person who created the Google Drive/Box/OneDrive/Dropbox folder will cause everyone to lose data or access if, for example, their university account is terminated. In Nextcloud, we are not at the mercy of individuals having full control over shared resources.  * We can grant secure access to anyone we need. Our federated authentication system allows basically anyone from anywhere to authenticate to our services; we support ORCID, Google, and GitHub in addition to thousands of academic institutions via CILogon. Instead of a flimsy "share-by-link", you can easily share files robustly with someone having a legit account.  * Nextcloud is free and open source, and there are official sync clients for *all platforms*, allowing you to work naturally with files synced to your desktop or even your phone. This integration is not always available on commerical services (e.g. there is no Box desktop sync client for Linux).  * You can share data securely and seamlessly with people on *completely independent Nextcloud servers* via the federated sharing system. --- ### Scheduling meetings and coordinating events ----------- You want to schedule weekly group meetings and other events. You want people to get automatic updates when the Zoom link changes or the time shifts due to a conflict. ---- #### Typical approach Email your invitees and agree on a time after a few Reply-All email exchanges. Everyone manually adds the event to their calendars independently. ---- #### MUSES approach * Create an event in the MUSES Nextcloud Calendar app in the "MUSES Collaboration" calendar that is already visible to the entire collaboration (or create a new calendar shared only with your team). * Add people to the attendees list by name (if they are a collaborator and thus have a Nextcloud account) or by email (if you want to invite external people). * People that have subscribed to the MUSES calendar(s) in their personal calendar apps will automatically see the new meeting appear. External invitees will receive an email with a convenient `.ics` file they can use to add/update the event to their personal calendar. ---- #### Why * If meeting details change, you only have to change the calendar event **once** in the MUSES calendar event and everyone will automatically get the update, including the external attendees who will receive an email notification. If you had manually coordinated by email, you would have to email everyone yourself (hoping that you don't miss anyone who may have been invited separately by someone else), and then each of them would have to diligently update their personal calendar events.  * By sharing your event in a MUSES calendar, others in the collaboration have the ability to see what is going on and may be inspired to engage with your team.  * Even if the calendar is more private to your team, there are other benefits. It is trivial to onboard new team members because you don't have to manually hunt through and forward emails to ensure they know about all your meetings and events. --- ### Discussing a new idea and brain-storming designs ----------- You have a new idea for how to interpolate calculated EoS values, and you need the input from collaborators both in the USA and in Japan. You know hashing things out in real-time meetings is difficult to schedule so you take an async approach. ---- #### Typical approach **Write an email to everyone who might want to participate** in the discussion of your new idea. This could be dozens of people. A few people Reply-All that day with feedback and ideas. ---- #### MUSES approach **Post your idea as a forum topic** in the appropriate category (so that people watching that category passively see it), and @mention specific people or groups to **ensure that they will get an email notification** to start the discussion. ---- #### Why * **There is no way for people to effectively opt-out of an email-based discussion**. If someone was included in the original email who actually has nothing to contribute and does not want the noise in their inbox, they have few good options. They can interrupt to ask that people stop including them in Reply-Alls, which is not guaranteed to happen, or they can make some special email filter that precludes them reentering the conversation.  * A forum topic solves this problem by allowing **fine-grained control over notifications**. For example, you can mute a topic yet still receive notifications when someone @mentions you specifically.  * **Email exchanges confuse chronology** because for example the third response in the exchange could be replying to the original message, while the second response is responding to the first response.  * **A forum topic has a chain of posts that make it easy to follow** who posted when and to what they are responding. It is also easy to "Reply as a linked topic" if you generate a tangential discussion that needs to be separated but that you want connected to the original topic, or you can reply to a post as a private message to one or more people. --- ### Capturing meeting notes and action items ----------- You have lively discussions during your weekly team meetings about many technical topics. Some of the discussion you understand well, but some you do not follow so easily. By the end of the meeting, you want a set of notes with the salient ideas and conclusions from the discussion, and you also want to know what you are concretely supposed to get done before the next meeting. ---- #### Typical approach Designate one person to be the note taker. After the meeting, the note taker emails the participants the meeting notes. Discussion about the notes and identified tasks takes place as Reply-All email exchanges with many copies and iterations of the notes evolving in each message. ---- #### MUSES approach Create a forum topic for that week's meeting notes, switching the post to "wiki" mode. Share a link to a HedgeDoc note that participants can collaboratively edit in real-time during the meeting. Drag and drop images and links to external resources into the notes as needed. After the meeting, copy the HedgeDoc note contents into the forum post. Subsequent edits can be made by anyone, and discussion about them can occur in replies within the topic. ---- #### Why * Editing notes in a collaborative real-time document tool like HedgeDoc relieves the burden on an individual note taker and makes it more likely the notes will be comprehensive and accurate by the end of the meeting.  * Keeping the discussion close to the notes makes it easy to iterate on ideas generated during the meeting that need follow-up. There is only one "source of truth" instead of scattered email, and  * people who were not even in the meeting can be pulled naturally into the conversation if needed by @mentioning them, where they will see things in context. --- ### Code development ----------- You are developing code with your team for your EoS module. You have weekly meetings about what progress you have made and what needs to be done next. ---- #### Typical approach Work on a copy of the code on your local workstation, installing dependencies as needed. Share what you have in meetings by sharing your screen virtually or in person. Receive and share updated versions of individual files as necessary by emailing them back and forth among your team. Explain in words at the meeting what you are working on and when you think you have completed a task. ---- #### MUSES approach * Store all code in a git repository hosted on an external service like GitLab.com. Grant privileged access to the team members so that they can push and pull to the repo.  * Identify concrete, manageable development tasks and capture them as Issues. Team members are passively notified when you comment on these issues with progress, where you reference specific commits that accomplish objectives, tightly coupling your task progress to code changes.  * Team members pull from the shared repo frequently to ensure all members are working with the same synchronized code.  * Functionality is demonstrated in a Docker container to guarantee a consistent operating environment in which team members execute the code. ---- #### Why * Version-control systems like git are an essential tool in modern software development, especially when your software will be integrated as a component in a larger software system developed by others. * Using a remote git hosting service like GitLab.com provides a powerful suite of collaborative and processing tools in addition to the basic backup functionality you need to protect against data loss. We will discuss this in-depth in my talk tomorrow! --- ## Conclusions ----------- Working with other people effectively in a collaboration like ours requires **learning new ways of doing things**. Sometimes it takes **a little extra effort** "up front" to **make things easier later** on. **Be proactive**. Ask yourself how your approach to a particular task will help others in the collaboration. If something does not work the way you want, or you do not know how to use a tool/service, **do not hesitate to contact us for help**. We will find a solution. ---- ## thank you Any more questions? <style> .reveal h3 { font-size: 4rem; font-weight: 500; margin-bottom: 6rem; //padding-bottom: 1rem; //border-bottom: 1px solid gray; } .reveal h4 { font-size: 3rem; font-weight: 500; margin-bottom: 5rem; } .reveal { font-size: 2.5rem; } .reveal a { text-decoration: underline; } /* p,ul { padding-left: 10%; padding-right: 10%; } */ </style>