Dear Yawi,
We detected a similar error due to incompatibility of TensorFlow versions.
The implementation was tested with TensorFlow 2.7.0. Please, could you
check that you are using this version?
You can use the following command to install all dependencies from scratch
(Python 3.9):
*# pip install notebook==6.4.11 PyYAML==6.0 tensorflow==2.7 networkx==2.8.1
matplotlib==3.5.2 astropy==5.1*
Best regards,
José Suárez-Varela
------------------------
Postdoctoral Researcher
Barcelona Neural Networking center (BNN-UPC)
Universitat Politècnica de Catalunya
El vie, 5 ago 2022 a las 21:31, yawi yoga (<yawiyoga(a)gmail.com>) escribió:
Hi,
After changing scheduling policy to WFQ in the training data, the training
script gives the following error when trying to train a model:
Node:
'gradients/rnn_13/TensorArrayUnstack/TensorListFromTensor_grad/TensorListStack'
Operation expected a list with 1 elements but got a list with 2 elements.
[[{{node
gradients/rnn_13/TensorArrayUnstack/TensorListFromTensor_grad/TensorListStack}}]]
[Op:__inference_train_function_32315]
I created the data using quickstart notebook, the only change is replacing
this line in generate_topology():
G.nodes[i]["schedulingPolicy"] = "FIFO"
with this
G.nodes[i]["schedulingPolicy"] = "WFQ"
G.nodes[i]["schedulingWeights"] = '25,25,50'
The docker runs OK on this data but I can't train a model on it due to the
above error.
If the policies are FIFO, the training runs without issues.
Could you please let me know how to resolve this?
Thanks,
Yawi
----- here is the full traceback for your reference:
INFO: Starting training from scratch...
Epoch 1/20
Traceback (most recent call last):
File "/home/yawi/dev/GNNetworkingChallenge/api_test.py", line 23, in
<module>
main(data_dir, final_evaluation=True)
File "/home/yawi/dev/GNNetworkingChallenge/RouteNet_Fermi/__init__.py",
line 101, in main
model.fit(ds_train,
File
"/home/yawi/.conda/envs/gnnch/lib/python3.9/site-packages/keras/utils/traceback_utils.py",
line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File
"/home/yawi/.conda/envs/gnnch/lib/python3.9/site-packages/tensorflow/python/eager/execute.py",
line 54, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph
execution error:
Detected at node
'gradients/rnn_13/TensorArrayUnstack/TensorListFromTensor_grad/TensorListStack'
defined at (most recent call last):
File "/home/yawi/dev/GNNetworkingChallenge/api_test.py", line 23, in
<module>
main(data_dir, final_evaluation=True)
File
"/home/yawi/dev/GNNetworkingChallenge/RouteNet_Fermi/__init__.py", line
101, in main
model.fit(ds_train,
File
"/home/yawi/.conda/envs/gnnch/lib/python3.9/site-packages/keras/utils/traceback_utils.py",
line 64, in error_handler
return fn(*args, **kwargs)
File
"/home/yawi/.conda/envs/gnnch/lib/python3.9/site-packages/keras/engine/training.py",
line 1409, in fit
tmp_logs = self.train_function(iterator)
File
"/home/yawi/.conda/envs/gnnch/lib/python3.9/site-packages/keras/engine/training.py",
line 1051, in train_function
return step_function(self, iterator)
File
"/home/yawi/.conda/envs/gnnch/lib/python3.9/site-packages/keras/engine/training.py",
line 1040, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))
File
"/home/yawi/.conda/envs/gnnch/lib/python3.9/site-packages/keras/engine/training.py",
line 1030, in run_step
outputs = model.train_step(data)
File
"/home/yawi/.conda/envs/gnnch/lib/python3.9/site-packages/keras/engine/training.py",
line 893, in train_step
self.optimizer.minimize(loss, self.trainable_variables, tape=tape)
File
"/home/yawi/.conda/envs/gnnch/lib/python3.9/site-packages/keras/optimizers/optimizer_v2/optimizer_v2.py",
line 537, in minimize
grads_and_vars = self._compute_gradients(
File
"/home/yawi/.conda/envs/gnnch/lib/python3.9/site-packages/keras/optimizers/optimizer_v2/optimizer_v2.py",
line 590, in _compute_gradients
grads_and_vars = self._get_gradients(tape, loss, var_list, grad_loss)
File
"/home/yawi/.conda/envs/gnnch/lib/python3.9/site-packages/keras/optimizers/optimizer_v2/optimizer_v2.py",
line 471, in _get_gradients
grads = tape.gradient(loss, var_list, grad_loss)
Node:
'gradients/rnn_13/TensorArrayUnstack/TensorListFromTensor_grad/TensorListStack'
Operation expected a list with 1 elements but got a list with 2 elements.
[[{{node
gradients/rnn_13/TensorArrayUnstack/TensorListFromTensor_grad/TensorListStack}}]]
[Op:__inference_train_function_32315]
Process finished with exit code 1
_______________________________________________
Challenge2022 mailing list
Challenge2022(a)mail.bnn.upc.edu
https://mail.bnn.upc.edu/cgi-bin/mailman/listinfo/challenge2022