I have a set of JSON files stored in a folder;the folder structure is as follows:
Main_folder----F1-----I1------X.json-----I2-------X.json......In all, the main folder contains 36 subfolders which in turn contain 10 subfolders each having a JSON file.
The JSON file is a dictionary of a listsX={'a':[contains 73 items each of which is a dictionary that represents a 2x3 matrix],'b':[contains 73 items each of which is a dictionary]}
I want to read these JSON files, get the 2x3 matrices and store them in .npy files.
My current code does that but it ends up saving the same npy file multiple times in the folders.For example, X_0.npy should be stored once in Main_folder/F1/I1, but that same file is being saved in other folders also.
import os, json,codecsimport numpy as npfrom pathlib import Pathdef read_pose_from_json(json_file): #read json file and obtain 73x6 pose array with open(json_file) as f1: json1_str=f1.read() json1_data=json.loads(json1_str) data=json1_data.copy() l1=data['world_coordinate'] pose_list=[] for i in range(len(l1)): X=list(l1[i].values())[0] X=np.array(X).ravel() pose_list.append(X) return np.array(pose_list)path_to_json='blender_files/'paths=[]paths1=[]#For all json files ,read them obtain pose array(73x6) and store it in a npy file so that it can#later be accesedfor (dirpath,dirnames,filenames) in os.walk(path_to_json): paths+=[os.path.join(dirpath,file) for file in filenames if file.startswith('bones_') and file.endswith('.json')] for i in range(len(paths)): pose_array=read_pose_from_json(paths[i]) new_pose_file=Path(paths[i]).stem+".npy" new_pose_file=os.path.join(dirpath,new_pose_file) with open(new_pose_file,'wb') as f: np.save(f,pose_array)What steps can I take to optimize this code, and ensure that redundant copies of the same file are not created?
I have enclosed a screenshot which shows what the code is currently doing:
In this image inside Model23/image7 there are multiple npy files being saved.
I was able to resolve the issue,I moved the loop
for i in range(len(pathfiles)outside the loop that recursively looks in all folders and sub-folders.so for now the correct code is
use os.walk() to list all json files and store them in list paththen iterate through paths list and obtain matrices and save themIf anyone else can suggest a better solution,that would be appreciated.
