Data wrangling. If at first glance this phrase conjures an image of a computer scientist cowboy, you wouldn’t be entirely wrong. What is data wrangling? Put simply, it is the process of finding and transforming different data to answer a question. Whether analyzing a former contract or compiling a list of attributes, data wrangling involves organizing data so it can be easily accessed by business professionals to answer their questions. While this simplified answer may suffice basic curiosity, there are more details surrounding this challenging task.
DATA WRANGLING DEFINED
The basic definition of data wrangling remains consistent with that above: the process of gathering, transforming and analyzing data to answer a question. However, the process is more complex, producing multiple data structures and requiring various steps to get to the final result. According to data analytics experts Elder Research, data wrangling commonly takes up 80% of analytics professionals’ time. This time-consuming process is undertaken to transform raw data into a tangible and easily digestible format that can then be used to optimize key decision-making by business professionals.
STRUCTURES AND STEPS OF DATA WRANGLING
According to Elder Research, data wrangling typically produces one of four data sets.
- Analytic Base Table: Used for machine learning where every row represents an entity (person, product, vehicle, etc.) and the columns consist of information about that entity during a point in time
- Denormalized Transactions: Transaction information used for business operations such as a line item in a specific order or a patient’s x-rays over time
- Time Series: Attributes about an entity over time
- Document Library: Documents used for analysis by text mining
To achieve one of these four structures, data analysts typically follow six steps, according to data wrangling software provider Trifacta.
1. DISCOVERING
The discovery process helps make sense of the data set being analyzed. For example, customer data can be analyzed and sorted by location, item purchased, promotion used, etc. Understanding the main goal of the data analysis and the data set you are searching in helps when compiling it into the final format.
2. STRUCTURING
Structuring organizes the data after discovery. The ultimate goal of this step is to compile the data into an easily understood format. This can involve separating information from one to multiple columns and/or creating new rows to help with the flow of the document.
3. CLEANING
Cleaning involves correcting errors or outliers in the data set and making it parallel. For example, ensuring all state names are consistent — WA. vs. Washington vs. Wash.— and checking that every column or row has a value. This helps increase the quality of the data.
4. ENRICHING
After cleaning, it’s necessary to reflect on the data and find other data sets that may make the final outcome better or enrich the data. Asking questions like “What other information would help inform the decision-making process?” or “Is there new data I can derive from what I currently have?”
5. VALIDATING
Much like step four, validating involves verifying consistency and quality. This can involve ensuring a parallel distribution of data or verifying accuracy.
6. PUBLISHING
Once the data is validated, it’s ready for publication. Analysts prepare the final document for use by business professionals. These individuals then make informed decisions based on the data they see.
PRACTICES AND TOOLS USED
Issues and challenges commonly arise during the data wrangling process, and analysts use different practices and tools to address any problems that occur. Common issues include the following.
- Gaining access to data
- Ensuring entities are clear
- Avoiding selection bias
- Finding missing values
- Creating a consistent format
Analysts use a number of tools to aid with these issues and properly format the final data set. According to data analytics and security company Varonis, the most popular and useful tools include DataWrangler, Tabula, and Python and Pandas.
DATA WRANGLER
This tool aids in data cleaning and transforming data into an easily understood table that can be exported into Excel and used to make real-world business decisions.
TABULA
Tabula is a tool that helps analysts gain access to data that would be otherwise time-consuming to get. Specifically, it takes PDF data and formats it into an Excel document. This saves the analysts from having to copy and paste each and every part of the data.
PYTHON AND PANDAS
Together, the Python coding language with the Pandas library allows analysts to merge and join large sets of data with one Python statement. This not only helps with ensuring entity clarity, but allows analysts to sort out the data more quickly and efficiently.
CAREERS AND RELATED SKILLS
Data wrangling is a time-consuming and challenging task, yet there are multiple careers paths and related skills associated with data wrangling.
DATA WAREHOUSE SPECIALIST
A data warehouse specialist acts as the liaison between programmers, data architects and analysts. This role helps manipulate and combine data, ensures data is managed correctly and performs technical administration duties. The median salary is $107,814, according to the latest figures from the U.S. Bureau of Labor Statistics (BLS)
DATABASE ADMINISTRATORS
Database Administrators manage software that stores and organizes large data sets such as customer shipping records. They ensure that data analysts can easily find the information they need. The median salary is $93,750, according to the BLS.
DATABASE WAREHOUSE ANALYST
A database warehouse analyst helps coordinate and design database architecture and researches business information needs. The BLS places the median salary for this role at $64,912.
DATA WRANGLING RELATED SKILLS
Specific skills such as coding, math, communication, data visualization and machine learning are needed to best perform data wrangling. Coding is necessary to find and organize data. Math, specifically statistics, helps with data processing, modeling and featuring visualization. Communication involves explaining the data set to others who may have limited data knowledge. Data visualization includes knowing the data type, visualizing what format to convey it in and mapping that data. Finally, machine learning involves being able to understand machine framework.
PURSUE A CAREER IN INFORMATION MANAGEMENT AT THE UNIVERSITY OF WASHINGTON
Additional education helps individuals gain the skills to excel in a key field such as data wrangling and grow their careers in information management.
Visit the University of Washington’s Master of Science in Information Management (MSIM) program page to learn more about pursuing a career in information management.
During the program, students build their professional network, gain industry experience and incorporate real-world examples into their studies while learning how to create a positive change in the field. Expand your information management career possibilities at the University of Washington.
SOURCES
Elder Research, “What is Data Wrangling and Why Does it Take so Long?”
Salary.com, “Average Salary for Jobs with Data Wrangling Skills”
Trifacta, “Data Wrangling Tools to Help Business Save Time”